Use motion data to generate higher resolution images

ABSTRACT

Techniques for using motion data to generate a high resolution output color image from multiple images having sparse color information are disclosed. A camera generates multiple images. The camera&#39;s sensor is configured to have a sparse Bayer pattern. While the camera is generating the images, IMU data for each image is acquired. The IMU data indicates a corresponding pose the camera was in while the camera generated each image. The images and the IMU data are fed as input into a motion model. The motion model performs temporal filtering on the images and uses the IMU data to generate a red-only image, a green-only image, and a blue-only image. A high resolution output color image is generated by combining the red-only image, the green-only image, and the blue-only image.

BACKGROUND

Mixed-reality (MR) systems, which include virtual-reality (VR) andaugmented-reality (AR) systems, have received significant attentionbecause of their ability to create truly unique experiences for theirusers. For reference, conventional VR systems create completelyimmersive experiences by restricting their users' views to only virtualenvironments. This is often achieved through the use of a head mounteddevice (HMD) that completely blocks any view of the real world. As aresult, a user is entirely immersed within the virtual environment. Incontrast, conventional AR systems create an augmented-reality experienceby visually presenting virtual objects that are placed in or thatinteract with the real world.

As used herein, VR and AR systems are described and referencedinterchangeably. Unless stated otherwise, the descriptions herein applyequally to all types of MR systems, which (as detailed above) include ARsystems, VR reality systems, and/or any other similar system capable ofdisplaying virtual content.

An MR system may also employ different types of cameras in order todisplay content to users, such as in the form of a passthrough image. Apassthrough image or view can aid users in avoiding disorientationand/or safety hazards when transitioning into and/or navigating withinan MR environment. An MR system can present views captured by cameras ina variety of ways. The process of using images captured by world-facingcameras to provide views of a real-world environment creates manychallenges, however.

To improve the quality of the images that are displayed to a user, someMR systems perform what is called “temporal filtering.” Temporalfiltering refers to the process by which the system combines data thatis captured over multiple timepoints in order to generate a particularoutput. In other words, the system essentially stacks multiple images ontop of one another and combines them in a manner so as to produce anaggregated image having an improved quality.

For instance, in the MR system scenario, the system combines image dataof consecutively captured images in order to generate an improvedoutput. As an example, consider a low light scenario. Because of the lowlight, each individual image might be capable of providing only alimited amount of image data. By combining the data from multipleconsecutively captured images, however, the system (e.g., by combiningthe data from all of those images via temporal filtering) can produce asuitable output image. In this sense, the process of temporal filteringinvolves capturing multiple image frames over a period of time and thencombining the image data from those frames to produce an output frame,resulting in a scenario where the output frame is actually anaggregation of multiple input frames.

Various challenges occur when performing temporal filtering, however.For example, so-called “ghosting effects” can result if an object in thescene or environment is moving while the system captures the multipleconsecutive images. More particularly, ghosting occurs when an object orimage artifact has a trail of pixels that follow the object (e.g., aform of motion blur). This trail of pixels occurs because the object isat different locations while the multiple consecutive images are beinggenerated, and those different locations are then reflected in the finalcomposite image.

Another challenge occurs when the camera itself undergoes movement whileit is generating the images. Movements of the camera can also skew thetemporal filtering process. It may be the case that the camera is movingin non-MR system scenarios, such as perhaps in vehicles. Accordingly,there are other technical areas where these challenges occur. Yetanother challenge relates to the quality or resolution of the finaloutput image. In view of these challenges, as well as others, there is asubstantial need to improve the temporal filtering process.

The subject matter claimed herein is not limited to embodiments thatsolve any disadvantages or that operate only in environments such asthose described above. Rather, this background is only provided toillustrate one exemplary technology area where some embodimentsdescribed herein may be practiced.

BRIEF SUMMARY

Embodiments disclosed herein relate to systems, devices, and methods forusing motion data to generate a high resolution output color image frommultiple images having sparse color information.

In some embodiments, a camera generates images. A sensor of the camerais configured to have a sparse Bayer pattern comprising one or more redpixels, one or more green pixels, one or more blue pixels, and aplurality of monochrome pixels. While the camera is generating theimages, the embodiments acquire corresponding IMU data for each of theimages. The IMU data indicates a corresponding pose the camera was inwhile the camera generated each image. The embodiments also feed theimages and the IMU data into a motion model. The motion model performstemporal filtering on the images and uses the IMU data to generate ared-only image, a green-only image, and a blue-only image. A highresolution output color image is generated by combining the red-onlyimage, the green-only image, and the blue-only image.

In some embodiments, the embodiments acquire supplemental informationthat reflects a corresponding pose the camera was in while the cameragenerated the images. This supplemental data can then be fed as inputinto the motion model along with the images in order to generate thered-only, green-only, and blue-only images.

In some embodiments, the motion model is configured to generate thered-only image by isolating red pixels included in the images fromnon-red pixels and by populating the red-only image with the red pixelsby placing each respective red pixel at a corresponding red-only imagecoordinate within the red-only image. The corresponding red-only imagecoordinate is determined using the IMU data. The motion model is furtherconfigured to generate the green-only image by isolating green pixelsincluded in the images from non-green pixels and by populating thegreen-only image with the green pixels by placing each respective greenpixel at a corresponding green-only image coordinate within thegreen-only image. The corresponding green-only image coordinate isdetermined using the IMU data. The motion model is further configured togenerate the blue-only image by isolating blue pixels included in theimages from non-blue pixels and by populating the blue-only image withthe blue pixels by placing each respective blue pixel at a correspondingblue-only image coordinate within the blue-only image. The correspondingblue-only image coordinate is determined using the IMU data.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

Additional features and advantages will be set forth in the descriptionwhich follows, and in part will be obvious from the description, or maybe learned by the practice of the teachings herein. Features andadvantages of the invention may be realized and obtained by means of theinstruments and combinations particularly pointed out in the appendedclaims. Features of the present invention will become more fullyapparent from the following description and appended claims, or may belearned by the practice of the invention as set forth hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features can be obtained, a more particular descriptionof the subject matter briefly described above will be rendered byreference to specific embodiments which are illustrated in the appendeddrawings. Understanding that these drawings depict only typicalembodiments and are not therefore to be considered to be limiting inscope, embodiments will be described and explained with additionalspecificity and detail through the use of the accompanying drawings inwhich:

FIG. 1 illustrates an example of an HMD.

FIG. 2 illustrates how the HMD can generate an image and how, in somecases, that image might be noisy.

FIG. 3 illustrates an example timeline illustrating how temporalfiltering is performed.

FIG. 4 illustrates one example of a camera sensor having a sparse Bayerpattern.

FIG. 5 illustrates another example of a camera sensor having a sparseBayer pattern.

FIG. 6 illustrates various curves showing the quantum efficiencies of asensor having a Bayer pattern.

FIG. 7 illustrates how images can be generated over time, where thecamera can shift somewhat between each image capture.

FIG. 8 illustrates an example process in which pixel data isprogressively collected or built up over time.

FIG. 9 illustrates an example architecture that may be used to generatea high resolution output color image.

FIG. 10 illustrates how a super resolution RGB image can be generated.

FIG. 11 illustrates a flowchart of an example process for generating ahigh resolution output image using motion data.

FIG. 12 illustrates how various different images can be generated.

FIG. 13 illustrates another flowchart of an example process forgenerating a super high resolution output image.

FIG. 14 illustrates an example computer system that may be configured toperform any of the disclosed operations.

DETAILED DESCRIPTION

Embodiments disclosed herein relate to systems, devices, and methods forusing motion data to generate a high resolution output color image frommultiple images having sparse color information.

In some embodiments, a camera generates images. A sensor of the camerais configured to have a sparse Bayer pattern comprising one or more redpixels, one or more green pixels, one or more blue pixels, and aplurality of monochrome pixels. While the camera is generating theimages, the embodiments acquire corresponding IMU data for each of theimages. The IMU data indicates a corresponding pose the camera was inwhile the camera generated each image. The embodiments also feed theimages and the IMU data into a motion model. The motion model performstemporal filtering on the images and uses the IMU data to generate ared-only image, a green-only image, and a blue-only image. A highresolution output color image is generated by combining the red-onlyimage, the green-only image, and the blue-only image.

As used herein, “super resolution” refers to any resolution thatincludes or is higher than full HD. As used herein, “sparse resolution”refers to any resolution that includes or is lower than SXGA.

Examples Of Technical Benefits, Improvements, And Practical Applications

The following section outlines some example improvements and practicalapplications provided by the disclosed embodiments. It will beappreciated, however, that these are just examples only and that theembodiments are not limited to only these improvements.

The disclosed embodiments bring about numerous benefits, advantages, andpractical applications to the technical field. That is, the embodimentsprovide improvements to the technical field of temporal filtering andgenerating images. For instance, the disclosed embodiments provide forthe ability for a system to operate in sub-optimal conditions, such aslow light conditions, yet still be able to produce high resolution fullcolor output images. To illustrate, the disclosed principles can bepracticed in low luminance environments where the camera sensors mightpossibly detect less than 1 photon per image frame (aka image). Despitethese sub-par conditions, the embodiments beneficially achieve improvedoutput signals by temporally combining multiple frames together.

The embodiments also beneficially reduce or entirely eliminate ghostingeffects, which might otherwise occur in traditional systems when objectsin the scene are moving. The principles can also be implemented evenwhere there are high levels of noise in the generated images, or whenthere is a low signal to noise ratio for the imagery. That is, theprinciples can be practiced in a broad range of conditions.

The embodiments use a camera sensor having a sparse Bayer pattern in anew and unique manner in order to generate a high resolution full coloroutput image. To do so, the embodiments utilize a motion model toisolate red, green, blue, and texture pixel data from the initial cameraimages. The motion model then generates respective red-only, green-only,and blue-only color images as well as a high resolution texture map(i.e. a monochrome image). The respective single-color images aregenerated using a temporal filtering process in which pixel data frommultiple images is extracted and aggregated together.

Notably, because the camera moved while the camera was generating thevarious images, different portions of the environment are represented or“covered” in the images using the isolated color data. For instance, atone point in time, a red pixel “covers” a first part of the environmentwhile at a second point in time, the red pixel “covers” a second part ofthe environment. Over time, a large majority of the environment willeventually be “covered” using the red pixel due to the movement of thecamera. The embodiments aggregate the red pixel data, which wascollected over time, to generate the red-only image. Similar operationsare performed for the green-only image, the blue-only image, and themonochrome/grayscale image. The resulting images can be higherresolution than the initial camera images.

With these images and map, the embodiments can then beneficially combinethose images and map in a manner so as to generate a high resolutionfull color output image. Beneficially, the disclosed principles can bepracticed in other, non-MR system scenarios, such as perhaps in thecontext of a Accordingly, these and numerous other benefits will now bedescribed throughout the remaining portions of this disclosure.

Example MR Systems And HMDs

Attention will now be directed to FIG. 1 , which illustrates an exampleof a head mounted device (HMD) 100. HMD 100 can be any type of MR system100A, including a VR system 100B or an AR system 100C. It should benoted that while a substantial portion of this disclosure is focused onthe use of an HMD, the embodiments are not limited to being practicedusing only an HMD. That is, any type of camera system can be used, evencamera systems entirely removed or separate from an HMD. As such, thedisclosed principles should be interpreted broadly to encompass any typeof camera use scenario. Some embodiments may even refrain from activelyusing a camera themselves and may simply use the data generated by acamera. For instance, some embodiments may at least be partiallypracticed in a cloud computing environment.

HMD 100 is shown as including scanning sensor(s) 105 (i.e. a type ofscanning or camera system), and HMD 100 can use the scanning sensor(s)105 to scan environments, map environments, capture environmental data,and/or generate any kind of images of the environment (e.g., bygenerating a 3D representation of the environment or by generating a“passthrough” visualization). Scanning sensor(s) 105 may comprise anynumber or any type of scanning devices, without limit.

In accordance with the disclosed embodiments, the HMD 100 may be used togenerate a passthrough visualizations of the user's environment. As usedherein, a “passthrough” visualization refers to a visualization thatreflects the perspective of the environment from the user's point ofview. To generate this passthrough visualization, the HMD 100 may useits scanning sensor(s) 105 to scan, map, or otherwise record itssurrounding environment, including any objects in the environment, andto pass that data on to the user to view. As will be described shortly,various transformations may be applied to the images prior to displayingthem to the user to ensure the displayed perspective matches the user'sexpected perspective.

To generate a passthrough image, the scanning sensor(s) 105 typicallyrely on its cameras (e.g., head tracking cameras, hand tracking cameras,depth cameras, or any other type of camera) to obtain one or more rawimages (aka “texture images”) of the environment. In addition togenerating passthrough images, these raw images may also be used todetermine depth data detailing the distance from the sensor to anyobjects captured by the raw images (e.g., a z-axis range ormeasurement). Once these raw images are obtained, then a depth map canbe computed from the depth data embedded or included within the rawimages (e.g., based on pixel disparities), and passthrough images can begenerated (e.g., one for each pupil) using the depth map for anyreprojections, if needed. A passthrough image can be generated as aresult of performing temporal filtering on multiple consecutivelygenerated images.

From the passthrough visualizations, a user will be able to perceivewhat is currently in his/her environment without having to remove orreposition the HMD 100. Furthermore, as will be described in more detaillater, the disclosed passthrough visualizations can also enhance theuser's ability to view objects within his/her environment (e.g., bydisplaying additional environmental conditions that may not have beendetectable by a human eye). As used herein, a so-called “overlaid image”can be a type of passthrough image.

It should be noted that while the majority of this disclosure focuses ongenerating “a” passthrough image, the embodiments actually generate aseparate passthrough image for each one of the user's eyes. That is, twopassthrough images are typically generated concurrently with oneanother. Therefore, while frequent reference is made to generating whatseems to be a single passthrough image, the embodiments are actuallyable to simultaneously generate multiple passthrough images.

In some embodiments, scanning sensor(s) 105 include visible lightcamera(s) 110, low light camera(s) 115, thermal imaging camera(s) 120,potentially ultraviolet (UV) camera(s) 125, potentially a dotilluminator 130, and even an infrared camera 135. The ellipsis 140demonstrates how any other type of camera or camera system (e.g., depthcameras, time of flight cameras, virtual cameras, depth lasers, etc.)may be included among the scanning sensor(s) 105.

As an example, a camera structured to detect mid-infrared wavelengthsmay be included within the scanning sensor(s) 105. As another example,any number of virtual cameras that are reprojected from an actual cameramay be included among the scanning sensor(s) 105 and may be used togenerate a stereo pair of images. In this manner, the scanning sensor(s)105 may be used to generate the stereo pair of images. In some cases,the stereo pair of images may be obtained or generated as a result ofperforming any one or more of the following operations: active stereoimage generation via use of two cameras and one dot illuminator (e.g.,dot illuminator 130); passive stereo image generation via use of twocameras; image generation using structured light via use of one actualcamera, one virtual camera, and one dot illuminator (e.g., dotilluminator 130); or image generation using a time of flight (TOF)sensor in which a baseline is present between a depth laser and acorresponding camera and in which a field of view (FOV) of thecorresponding camera is offset relative to a field of illumination ofthe depth laser.

The visible light camera(s) 110 are typically stereoscopic cameras,meaning that the fields of view of the two or more visible light camerasat least partially overlap with one another. With this overlappingregion, images generated by the visible light camera(s) 110 can be usedto identify disparities between certain pixels that commonly representan object captured by both images. Based on these pixel disparities, theembodiments are able to determine depths for objects located within theoverlapping region (i.e. “stereoscopic depth matching” or “stereo depthmatching”). As such, the visible light camera(s) 110 can be used to notonly generate passthrough visualizations, but they can also be used todetermine object depth. In some embodiments, the visible light camera(s)110 can capture both visible light and IR light.

In some embodiments, the visible light camera(s) 110 and the low lightcamera(s) 115 (aka low light night vision cameras) operate inapproximately the same overlapping wavelength range. In some cases, thisoverlapping wavelength range is between about 400 nanometers and about1,100 nanometers. Additionally, in some embodiments these two types ofcameras are both silicon detectors.

One distinguishing feature between these two types of cameras is relatedto the illuminance conditions or illuminance range(s) in which theyactively operate. In some cases, the visible light camera(s) 110 are lowpower cameras and operate in environments where the illuminance isbetween about 10 lux and about 100,000 lux, or rather, the illuminancerange begins at about 10 lux and increases beyond 10 lux. In contrast,the low light camera(s) 115 consume more power and operate inenvironments where the illuminance range is between about 110 micro-luxand about 10 lux.

The thermal imaging camera(s) 120, on the other hand, are structured todetect electromagnetic radiation or IR light in the far-IR (i.e.thermal-IR) range, though some embodiments also enable the thermalimaging camera(s) 120 to detect radiation in the mid-IR range. Toclarify, the thermal imaging camera(s) 120 may be a long wave infraredimaging camera structured to detect electromagnetic radiation bymeasuring long wave infrared wavelengths. Often, the thermal imagingcamera(s) 120 detect IR radiation having wavelengths between about 8microns and 14 microns. These wavelengths are also included in the lightspectrum(s). Because the thermal imaging camera(s) 120 detect far-IRradiation, the thermal imaging camera(s) 120 can operate in anyilluminance condition, without restriction.

The HMD 100 can also be equipped with an inertial measurement unit(IMU), as shown by IMU 145. The IMU 145 measures forces, angular rates,and orientation using a combination of accelerometers, gyroscopes, andmagnetometers. The IMU 145 produces IMU data, which can be used by thedisclose embodiments.

Accordingly, as used herein, reference to “visible light cameras”(including “head tracking cameras”), are cameras that are primarily usedfor computer vision to perform head tracking. These cameras can detectvisible light, or even a combination of visible and IR light (e.g., arange of IR light, including IR light having a wavelength of about 850nm). In some cases, these cameras are global shutter devices with pixelsbeing about 3 μm in size. Low light cameras, on the other hand, arecameras that are sensitive to visible light and near-IR. These camerasare larger and may have pixels that are about 8 μm in size or larger.These cameras are also sensitive to wavelengths that silicon sensors aresensitive to, which wavelengths are between about 350 nm to 1100 nm.Thermal/long wavelength IR devices (i.e. thermal imaging cameras) havepixel sizes that are about 10 μm or larger and detect heat radiated fromthe environment. These cameras are sensitive to wavelengths in the 8 μmto 14 μm range. Some embodiments also include mid-IR cameras configuredto detect at least mid-IR light. These cameras often comprisenon-silicon materials (e.g., InP or InGaAs) that detect light in the 800nm to 2 μm wavelength range.

The disclosed embodiments may be structured to utilize numerousdifferent camera modalities. The different camera modalities include,but are not limited to, visible light or monochrome cameras, low lightcameras, thermal imaging cameras, and UV cameras.

It should be noted that any number of cameras may be provided on the HMD100 for each of the different camera types (aka modalities). That is,the visible light camera(s) 110 may include 1, 2, 3, 4, 5, 6, 7, 8, 9,10, or more than 10 cameras. Often, however, the number of cameras is atleast 2 so the HMD 100 can perform passthrough image generation and/orstereoscopic depth matching, as described earlier. Similarly, the lowlight camera(s) 115, the thermal imaging camera(s) 120, and the UVcamera(s) 125 may each respectively include 1, 2, 3, 4, 5, 6, 7, 8, 9,10, or more than 10 corresponding cameras.

In some scenarios, noise may be present in one of the images. Forinstance, in very low light conditions (e.g., 1.0 millilux or“starlight” environments), there might not be enough light photons inthe environment for the low light camera to generate a high qualityimage. Indeed, the resulting image generated by the low light camera maybe heavily corrupted with noise. When such conditions occur, theembodiments are beneficially able to perform temporal filtering. Moredetails on temporal filtering will be provided shortly.

By way of reference, however, it is beneficial to note the generalilluminance metrics for different scenarios. For instance, a brightsunny day typically has an ambient light intensity of around10,000-50,000 lux. An overcast day typically has an ambient lightintensity of around 1,000-10,000 lux. An indoor office typically has anambient light intensity of around 100-300 lux. The time of daycorresponding to twilight typically has an ambient light intensity ofaround 10 lux. Deep twilight has an ambient light intensity of around 1lux. As used herein, a so-called “low light environment” at leastcorresponds to any environment in which the ambient light intensity isat or below about 40 lux. The HMD has one or more sensors that areconfigured to determine the surrounding environment lux intensity. Thesesensors can be incorporated into or independent from the cameras and/orilluminators described herein.

When used in a very low light environment (e.g., about 1.0 millilux or“starlight” environments), the low light camera sensors attempt tocompensate for the low light condition by ramping up the camera's gain(e.g., digital gain, analog gain, or a combination of digital and analoggain). As a result of ramping up the camera sensor's gain, the resultingimage can be very noisy. In an effort to improve the quality of theimages, the embodiments perform temporal filtering.

Temporal Filtering

FIG. 2 shows an HMD 200, which is representative of the HMD 100 of FIG.1 . HMD 200 has generated an image 205 in a low light environment 210.As shown, the image 205 includes various different image data 215.Because the image 205 was generated in a low light scenario, however,there is a significant amount of noise 220 (e.g., the white dotsthroughout the image 205). If used by itself, the image 205 wouldprovide a generally poor quality image for presentation to a user. Withthat understanding, then, the embodiments are configured to performtemporal filtering. FIG. 3 provides some useful details.

FIG. 3 shows a timeline 300. At time T₀, an HMD 305 (which isrepresentative of the HMDs discussed thus far) generates an image 310that includes image data 315. Similar to the scenario presented in FIG.2 , the image 310 was generated in a low light environment. FIG. 3 alsonotes how, subsequent to time T₀, the HMD undergoes some amount ofmotion 320 or movement.

At time T₁, the HMD 305 then generates a second image 325, whichincludes image data 330. In accordance with the disclosed principles,the embodiments are able to use image 310 and image 325 to perform atemporal filtering 335 operation. The embodiments can also use motiondata 340 generated by an IMU to compensate for the motion 320 thatoccurred between time T₀ and T₁. That is, the embodiments can acquireIMU data 345, which details the angular position 350 and theacceleration 355 of the camera that generated the images 310 and 325.The angular position 350 details the orientation of the camera using athree degrees of freedom (DOF) basis, while the acceleration 355 detailswhether the camera is translating or moving.

A motion compensation operation generally involves modifying one pose tomatch a different pose. The different pose might be a predicted posebased on the motion data 340. As an example, image 310 reflected orembodied a first pose of the HMD 305 at time T₀. The HMD 305 thenshifted to a new position or pose, as shown by motion 320. The IMUcaptured the movement of the HMD 305 between time T₀ and T₁. The HMD 305can use the IMU data to predict a new pose of the HMD 305 at time T₁ (orperhaps even a later time). Based on this prediction, the HMD 305 canthen perform a motion compensation operation to transform the poseembodied in image 310 to reflect the predicted pose of the HMD 305 (asit will be at time T₁). At time T₁, the HMD 305 generates the image 325.The embodiments are able to compare the motion compensated pose (whichwas designed in an attempt to reflect the HMD 305's pose at time T₁)with the actual pose of the HMD 305 at time T₁. The level or amount ofdifference between those two poses reflects the accuracy of the motioncompensation. If there is no difference, then the motion compensationoperation was 100% successful. On the other hand, the larger the amountof difference, the worse the motion compensation performed. Measuring orcomparing the differences is primarily performed by comparing eachpixel's intensity level with one another. That is, a first pixel in themotion compensated image is identified, and a corresponding pixel in theimage 325 is identified. The intensity levels for these two pixels arecompared against one another. The resulting difference between those twovalues reflects the accuracy of the motion compensation operation.

Performing motion compensation is desirable because of the temporalfiltering process. Recall, the temporal filtering process essentiallystacks multiple images on top of one another and combines the data fromthose images to generate an enhanced image. For the stacking to workproperly, the poses in each of those different images should align withone another. Thus, the motion compensation operation is performed inorder to align the various different poses for the various differentimages.

As a result of performing the temporal filtering 335 operation, theembodiments are able to generate a temporally filtered image 360 thathas improved or enhanced image data 365 as compared to the image data315 and 330 of the previous images. Additionally, as will be describedmomentarily, not only do the embodiments beneficially compensate formotions of the camera, but the embodiments also beneficially reduce theimpact of ghosting effects 370 for objects that are moving in the scene.Notably, in some embodiments, an exposure setting 370 of the cameraremains unchanged while the camera generates the images.

Bayer Pattern Characteristics

FIG. 4 shows an example camera sensor 400, which can be implemented inany of the cameras mentioned thus far. This sensor 400 is configured tohave a sparse Bayer pattern 405. By “sparse,” it is meant that multipledifferent monochrome (i.e. grayscale) pixels are interweaved betweenvarious red, green, and blue pixels. To illustrate, the sparse Bayerpattern 405 includes a red pixel 410, a green pixel 415 (two greenpixels are present, but only one is labeled), and a blue pixel 420. Thesparse Bayer pattern 405 further includes multiple different monochromepixels, as shown by monochrome pixel 425. In this example illustration,the sensor 400 has 16 pixels. Here, one pixel is a red pixel, two pixelsare green pixels, one pixel is a blue pixel, and 12 pixels aremonochrome.

The embodiments are able to isolate the color data from the differentpixels using different channels per pixel, as shown by channel 430. Thatis, one channel (e.g., a red sensor channel) can be used to isolate thered pixel data, another channel (e.g., a green sensor channel) can beused to isolate the green pixel data, a different channel (e.g., a bluesensor channel) can be used to isolate the blue pixel data, and yet adifferent channel (e.g., a monochrome sensor channel) can be used toisolate the monochrome pixel data.

FIG. 5 shows another example implementation of a sparse Bayer pattern500. From this illustration, one will appreciate how the configurationand layout of the sparse Bayer pattern can be designed to fit variousdifferent criteria. Generally, however, to be “sparse,” the Bayerpattern is configured to have various monochrome pixels interweavedamong red, green, and blue pixels.

In the above example, the pattern included one red pixel, two greenpixels, and one blue pixel, but other numbers can be used as well. Forinstance, it might be the case that there are two red pixels, one greenpixel, and one blue pixel, or, alternatively, one red pixel, one greenpixel, and two blue pixels. With a larger grid layout (e.g., perhaps 5×5or 6×6), the patterns can include multiple red pixels, multiple greenpixels, and multiple blue pixels.

FIG. 6 shows an example chart, which is labeled quantum efficiency chartusing Bayer pattern 600. As the name suggests, this chart outlines thequantum efficiencies (QE) for the red, green, blue, and monochromepixels of the sensor having the Bayer pattern. To illustrate, the chartincludes the red QE 605, the green QE 610, the blue QE 615, and themonochrome QE 620. Notice, the curve for the monochrome QE 620 is moreexpansive than any of the other curves. Generally, when a Bayer patternis used, about half of the optical power or half of the incident photonsare lost because of the color filters used as a part of the Bayerpattern (i.e. a loss in color generation fidelity). One of the benefitsprovided by the disclosed principles relates to the ability to collectmore photons over time in order to build up a higher resolution and moreintense full color output image. In this sense, the embodimentsbeneficially use a camera sensor having a sparse Bayer pattern in a newand unique manner in order to generate a high resolution full colorimage.

Building Up An Image

Attention will now be directed to FIG. 7 , which illustrates acompilation of images 700, such as image 705 and image 710. Each of theimages was generated using the Bayer pattern mentioned earlier. That is,in accordance with the disclosed principles, the embodiments are able touse the previously described sensor (e.g., sensor 400 from FIG. 4 ) togenerate multiple images over at least a period of time. This period oftime can be any time period, without limit, provided that multipleimages are generated during that time period.

By way of further clarification, at one point in time, the sensor ispositioned at Position A and generates the image 705 (in this simplifiedexample, the image is a 4×4 pixel image). At a different point in time,the same sensor is at Position B and generates the image 710. FIG. 7shows how multiple different images can be generated over a period oftime and how the sensor can be positioned at different locations duringthat time period. Based on the configuration of the sensor, the imageswill have a particular resolution 715 (e.g., perhaps a VGA 640×480resolution).

The embodiments are able to track the various positions of the sensorwhile the sensor is generating the images. In some cases, the trackingprocess is performed using inertial measurement unit (IMU) data 720.That is, the IMU data 720 tracks and monitors the various positions ofthe sensor, such as Position A and Position B at their respective times,or timestamps. As will be described later, image correspondences orfeature matching can also be performed to track the position of thesensor over time.

If multiple pixels “cover” the same area of the environment over time,the embodiments can perform a number of operations. For instance, insome cases, the embodiments might elect to use the most recent versionof the pixel data and might discard the old or stale version of thepixel data. In some cases, the embodiments might perform an intensitycomparison between the older and the newer pixels. If there is adiscrepancy between the two pixels, then it might be the case that anobject in the environment has moved (which could cause ghostingeffects). The embodiments can then analyze the entire image to determinewhether an object has moved. If one of the pixels reflects a scenariowhere there is no object while the other pixel reflects a scenario wherethere is an object present, then the embodiments can elect to use theformer pixel, thereby eliminating possible ghosting effects.

In one scenario, pixels that represent (or “cover”) the same area of theenvironment can be given different weights. These weights can beassigned using a bilateral weight computation. For instance, adifference in intensity between the pixels can be computed. Then, afunction can be defined that converts the difference to a weight, wherethe function is based on a negative exponential. Larger differencesresult in a smaller weight while smaller differences result in a higherweight. The weights can be used to determine the influence each pixelhas when combining the pixel data to create subsequent images (i.e.so-called red-only, green-only, blue-only, or even monochrome images).

The embodiments are able to isolate the different color data from oneanother. That is, the red pixel color data can be isolated from thegreen, blue, and monochrome color data. As will be described in moredetail later, this isolation is performed using different color channelsfor the different pixels as well as using a motion model to interpretthe data.

FIG. 8 shows a progressive buildup 800 process, where image data iscollected over time. By progressively collecting color data over aperiod of time, the embodiments can then stack that color data (i.e.perform temporal filtering) and effectively “build” a resulting imagebased on the individually collected (over time) pixel data. That imageis built by performing the motion compensation operation describedearlier. Specifically, IMU (or other supplemental information) isacquired in order to determine the differences in pose from one image toanother. The poses can be aligned using the motion compensationoperation. With the aligned data, color information can then be “filledin” in order to generate an image.

To illustrate, FIG. 8 shows a first instance in time (i.e. time “A”). Attime A, the camera generated an image using a sensor having a sparseBayer pattern. Here, the discussion will focus only on the red pixeldata, but one will appreciate how the principles equally apply for thegreen, blue, and monochrome pixels. Using the sparse Bayer pattern, animage is generated and there is (at this point in time) data for asingle red pixel 805. The embodiments are able to use an IMU to trackthe position or pose 810 of the camera. The embodiments are also able tostore the information related to the red pixel 805.

At time “B,” another image is generated. In this example scenario, thecamera has slightly shifted in position, and the position of the redpixel is now slightly to the right of where it was previously. With theIMU data, motion compensation can be performed to determine the relativelocations between the two pixels. Now, the embodiments have data for twored pixels 815. That is, the same sensor pixel generated the data atboth times, but the embodiments are able to track the data over time,resulting in a scenario where now there are multiple units of data.

At time “C,” the camera has again shifted position, and the camera hasagain generated another image and again determined the relativelocations of the pixels. Now, the embodiments have data for threedifferent red pixels 820.

Later, at time “D,” the camera has shifted numerous times and numerousdifferent images have been generated. From FIG. 8 , one can observe howan entire image is being “filled in” with red pixel data over a periodof time. The red pixels 825 illustrate this progressive buildup of colordata.

Even later, at time “E,” the camera has shifted many times and manydifferent images have been generated. Now, it is the case that an entireimage can be populated with pixel data that has been collected over timeusing the motion compensated temporal filtering process describedearlier (but based on individualized pixel data, such as all red pixeldata from a particular image). The red pixels 830 illustrate thisprogressive buildup of color data over time.

By tracking the pose 810 information for the camera for each of theimages, the embodiments can generate a resulting image and can populatea particular pixel in that resulting image with the acquired pixelinformation. Notably, the embodiments can determine which coordinates toplace the pixel in the resulting image based on the pose 810 informationpreviously acquired and based on an analysis (i.e. the motioncompensation operation) performed by a motion model, which will bedescribed shortly. FIG. 8 illustrates how the embodiments can determinecoordinates 835 for the red pixel 805 based on the pose information.Similarly, other coordinates (e.g., coordinates 840, 845, 850, and 855)can be determined for the other pixels. These coordinates indicate wherein the resulting image the red pixel data will be placed.

Using a Motion Model to Generate Images

FIG. 9 shows an example architecture 900 that may be used to generate ahigh resolution full color output image from lower resolution sparsecolor input images. Initially, a set of pixel data 905 is acquired usingthe processes outlined earlier. This pixel data 905 includes intensity910 information for the pixels (e.g., from the monochrome pixels), andthe images that generated the pixel data 905 have a sparse resolution915 (e.g., perhaps VGA 640×480 resolution). The sparseness of the colordata or resolution may be due to sub-optimal lighting conditions, suchas perhaps low light conditions.

The pixel data 905 is fed as input into a motion model 920. In additionto the pixel data 905, supplemental information 925 is also fed as inputinto the motion model 920. This supplemental information 925 can includefeature matching 930 data (i.e. image correspondences) or IMU data 935.

The feature matching 930 data uses matches between feature points (akaimage correspondences) to align the various images that were generated(e.g., the pixel data 905). In other words, the embodiments can identifyimage correspondences between the various different images. Those imagecorrespondences can then be used to determine how to align the variousimages with one another using the motion model 920. When the featurematching 930 is used, it may be the case that the motion model 920 is ahomography motion model, a similarity transform motion model, or perhapsan affine motion model.

On the other hand, the IMU data 935 can also be used to determine how toalign the various images, or rather, to determine what the relativepositioning of the pixel data is to one another. When the IMU data 935is used, the motion model 920 can be any type of three-dimensionalrotational motion model.

In any event, the supplemental information 925 is used by the motionmodel 920 to identify relative alignments between the various imagesthat generated the pixel data 905. The pixel data 905 is fed into themotion model 920 using different channels for the different pixel types(e.g., red, green, blue, monochrome), and the motion model 920 is ableto analyze the incoming information in order to isolate the pixel datafrom one another.

That is, using the incoming information, the motion model 920 is able toisolate and store all of the red pixels (e.g., red pixel 940) from allof the non-red pixels. The motion model 920 is able to isolate and storeall of the green pixels from all of the non-green pixels. The motionmodel 920 is able to isolate and store all of the blue pixels from allof the non-blue pixels. Similarly, the motion model 920 is able toisolate and store all of the monochrome pixels from all of thenon-monochrome pixels.

FIG. 9 shows one example of the above isolation process. For instance,the motion model 920 is able to analyze the pixel data 905 and thesupplemental information 925 to identify the red pixels 940. Based onthe red pixels 940, the motion model 920 can then generate a red-onlyimage 945.

Notably, the pixel data in that image is arranged relative to oneanother based on the supplemental information 925, which was used by themotion model 920 to determine the relative positioning of the red pixeldata relative to one another. By way of further clarification, themotion model 920 used the supplemental information 925 to determine afirst image coordinate for a first red pixel in the red-only image 945.The motion model 920 used the supplemental information 925 to determinea second image coordinate for a second red pixel in the red-only image945. The motion model 920 determined placement locations (i.e. imagecoordinates) for all of the red pixels 940, where that placementlocation indicated where in the red-only image 945 any particular onered pixel will be placed.

Similar operations are performed to generate a green-only image 950, ablue-only image 955, and a monochrome image 960 (aka a high resolutiontexture map). These respective images represent motion compensated (i.e.aligned) stacked versions of the pixel data that has been collected.Stated differently, these respective images are composites of all therespective color information that has been collected, and that colorinformation is aligned relative to one another via the motion model 920using the supplemental information 925.

Notice, the resolution of the monochrome image 960 is higher (i.e. asrepresented by the increased number of “boxes”) than the resolutions ofthe other images. The monochrome image 960 has a higher resolutionbecause there are a higher number of monochrome pixels than any otherpixels. Consequently, more monochrome data (i.e. texture data) isacquired as compared to the amount of the color data. Further details onthis aspect will be provided later.

In some cases, the red, green, and blue-only images might also havehigher resolutions than the original images that generated the pixeldata 905 (e.g., higher resolution than VGA resolution), though that maynot always be the case. In some cases, the resolutions of the red-only,green-only, and blue-only images might be the same as the originalimages that generated the pixel data 905 (e.g., perhaps VGA 640×480resolution i.e. “sparse” resolution or perhaps SXGA 1280×960resolution). Typically, the resolution of the red-only, green-only, andblue-only images will be higher than the resolution of the originalcamera images due to the stacking effect of the temporal filteringprocess.

The embodiments are able to use the red-only image 945, the green-onlyimage 950, the blue-only image 955, and optionally the monochrome image960 to then generate a high resolution output (full) color image 965.That is, the color and texture information from those respective imagescan be merged together to generate the high resolution output colorimage 965.

In scenarios where there is missing color data, such as perhaps wherethere is a lack of pixel data 905 for certain areas of the environmentor scene (e.g., voids or gaps in the resulting red-only, green-only, orblue-only images), the embodiments can perform bilateral interpolation970 to fill in the missing areas with color data. The bilateralinterpolation 970 can be performed on any one or more of the red-onlyimage 945, the green-only image 950, the blue-only image 955, themonochrome image 960, or even the high resolution output color image965.

In one scenario, the monochrome image 960 can be used as an index orperhaps as a frame of reference to help clarify points of texture orboundary information when combining the color images. By using themonochrome image 960 for the merging process to create the highresolution output color image 965, the monochrome image 960 can also beused to enhance the resulting texture or intensity in the highresolution output color image 965.

The motion model 920 can perform the various combination processes inthe RGB color space or, alternatively, in the HSI (hue, saturation,intensity) space. That is, the color data can be used to populate the Hand S channels while the monochrome image 960 can be used to populatethe I channel. Accordingly, different techniques are available forcombining the color data.

FIG. 10 shows an example super resolution process 1000 where an outputimage having super resolution (e.g., “2560×2048” resolution or perhaps“2K” resolution or perhaps “full HD” and above) can be generated fromlower resolution input images (e.g., perhaps VGA 640×480 resolution orperhaps SXGA 1280×960 resolution). Although the disclosure hasexplicitly called out certain resolutions (e.g., VGA, SXGA, or superresolution), one will appreciate how other resolutions can be used aswell. Such examples include, but are not limited to, MCGA, QVGA, VGA,Super VGA, XGA, SXGA, UXGA, full HD, 2K, 4K, 8K, 16K, and so on withoutlimit. As mentioned earlier, as used herein, “super resolution” refersto any resolution that includes or is higher than full HD. As usedherein, “sparse resolution” refers to any resolution that includes or islower than SXGA.

Initially, the super resolution process 1000 includes acquiring an image1005 (e.g., such as the red-only, green-only, or blue-only imagesmentioned earlier). This image 1005 has a particular resolution 1010. Insome cases, the resolution 1010 is considered sparse. In some cases, theresolution 1010 is higher than the sparse resolution of the initialcamera images. Notably, the resolution 1010 is lower than the resolutionof a monochrome image.

Additionally, the super resolution process 1000 includes acquiring amonochrome image 1015 (e.g., monochrome image 960). Because of theincreased number of monochrome pixels, the resulting monochrome image1015 will have a relatively higher resolution 1020. In some cases, thisrelatively higher resolution 1020 (as compared to the resolution 1010)is a super resolution (e.g., 2560×2048 resolution).

In accordance with the disclosed principles, the image 1005 and themonochrome image 1015 are fed as inputs into the motion model 1025. Theimage 1005 is then up-sampled or interpolated (e.g., as shown byup-sample 1030) to cause the resolution 1010 of the image 1005 to matchthe resolution 1020 of the monochrome image 1015. In some cases, theup-sample 1030 can be a bilateral up-sampling process. This process canbe performed on all of the red-only, green-only, and blue-only images.

With the image 1005 (e.g., the red-only, green-only, and blue-onlyimages) now having a higher resolution, the embodiments can then combinethat image with the other up-sampled images (i.e. combine all of thered-only, green-only, and blue-only images) to generate a superresolution RGB image 1035, which has at least the resolution 1020 of themonochrome image 1015. That is, all of the red-only, green-only, andblue-only images can be up-sampled and can then be combined or mergedwith one another to generate the super resolution RGB image 1035.Further details on this process with be provided later when the methodflowcharts are presented.

Example Methods

The following discussion now refers to a number of methods and methodacts that may be performed. Although the method acts may be discussed ina certain order or illustrated in a flow chart as occurring in aparticular order, no particular ordering is required unless specificallystated, or required because an act is dependent on another act beingcompleted prior to the act being performed.

FIG. 11 shows a flowchart of an example method 1100 for using motiondata to generate a high resolution output color image from a pluralityof images having sparse color information. Method 1100 can beimplemented in the architecture 900 of FIG. 9 .

Initially, method 1100 includes an act (act 1105) of causing a camera togenerate images. Here, a sensor of the camera is configured to have asparse Bayer pattern comprising one or more red pixels, one or moregreen pixels, one or more blue pixels, and a plurality of monochromepixels. Notably, each of the images includes sparse color data andsparse intensity data. Furthermore, an exposure setting of the cameraremains unchanged while the camera generates the images.

While the camera is generating the images, act 1110 includes acquiringcorresponding inertial measurement unit (IMU) data for each of theimages. Consequently, a plurality of IMU data is also generated. The IMUdata for each image indicates a corresponding pose the camera was inwhile the camera generated each image.

In act 1115, the embodiments feed the images and the IMU data into amotion model. The motion model performs temporal filtering on the imagesand uses the IMU data to generate a red-only image, a green-only image,and a blue-only image.

Act 1115, which involves feeding the images and IMU data into a motionmodel, can include a number of sub-steps. For instance, to implement act1115, the motion model can be configured to perform the operationsoutlined in FIG. 12 . For instance, act 1115 can include acts 1200,1205, 1210, and 1215.

Act 1200 includes generating a red-only image by isolating red pixelsincluded in the images from non-red pixels. Act 1200 further involvespopulating the red-only image with the red pixels by placing eachrespective red pixel at a corresponding red-only image coordinate withinthe red-only image. Notably, the corresponding red-only image coordinateis determined using the IMU data.

Act 1205 includes generating a green-only image by isolating greenpixels included in the images from non-green pixels. Act 1205 furtherinvolves populating the green-only image with the green pixels byplacing each respective green pixel at a corresponding green-only imagecoordinate within the green-only image. The corresponding green-onlyimage coordinate is determined using the IMU data.

Act 1210 includes generating a blue-only image by isolating blue pixelsincluded in the images from non-blue pixels. Act 1210 further includespopulating the blue-only image with the blue pixels by placing eachrespective blue pixel at a corresponding blue-only image coordinatewithin the blue-only image. The corresponding blue-only image coordinateis determined using the IMU data.

Act 1215 includes generating a monochrome image by isolating monochromepixels included in the images from non-monochrome pixels. Act 1215further includes populating the monochrome image with the monochromepixels by placing each respective monochrome pixel at a correspondingmonochrome image coordinate within the monochrome image. Thecorresponding monochrome image coordinate is determined using the IMUdata.

The resolutions of the red-only, green-only, blue-only, and monochromeimages can optionally be higher than the resolutions of the originalimages that generated the pixel data. In some cases, the resolution ofthe monochrome image is higher than any of the other images.

As alternatives to acts 1110 and 1115, method 1100 can include acts 1120and 1125. That is, while the camera is generating the images, act 1120includes acquiring supplemental information that reflects acorresponding pose the camera was in while the camera generated eachimage. This supplemental information can include the imagecorrespondences (i.e. detected feature points) mentioned earlier.

Act 1125 involves feeding the images and the supplemental informationinto a motion model. The motion model performs temporal filtering on theimages and uses the supplemental information to generate a red-onlyimage, a green-only image, and a blue-only image.

Act 1130 then involves generating a high resolution output color imageby combining the red-only image, the green-only image, and the blue-onlyimage. In some cases, the high resolution output color image isgenerated by merging intensities provided by the monochrome image withcolors provided by the red-only image, the green-only image, and theblue-only image. Optionally, an intensity of the high resolution outputcolor image can be higher than an intensity of any of the red-onlyimage, the green-only image, or the blue-only image. The process ofcombining the red-only image, the green-only image, and the blue-onlyimage can be performed by merging red pixels from the red-only imagewith corresponding green pixels from the green-only image and withcorresponding blue pixels from the blue-only image.

FIG. 13 illustrates a flowchart of an example method 1300 for generatinga high resolution full color output image from lower resolution sparsecolor input images. Method 1300 can be performed using the architecture900 of FIG. 9 . Method 1300 is somewhat similar to method 1100, butmethod 1300 includes a few different operations in order to generate asuper resolution full color output image. These additional operationsgenerally involve various different up-sampling techniques.

Similar to method 1100, there is an act (act 1305) of causing a camerato generate multiple images. Here, a sensor of the camera is configuredto have a sparse Bayer pattern. In some cases, the number of monochromepixels can be more than a sum of the red, green, or blue pixels. Each ofthese images includes sparse color data and sparse intensity data.Furthermore, an exposure setting of the camera remains unchanged whilethe camera generates these images. In some cases, the camera can be alow light camera; consequently, the images can be multiple low lightimages.

While the camera is generating the images, act 1310 includes acquiringcorresponding inertial measurement unit (IMU) data for each of theimages. Consequently, a plurality of IMU data is also generated. The IMUdata for each image indicates a corresponding pose the camera was inwhile the camera generated that respective image. That is, the IMUtracks different locations of the camera sensor over the period of time.

In act 1315, the embodiments feed the images and the IMU data into amotion model. The motion model performs temporal filtering on the imagesand uses the IMU data to generate a red-only image, a green-only image,a blue-only image, and a monochrome image.

As alternatives to acts 1310 and 1315, method 1300 can include acts 1320and 1325. For instance, while the camera is generating the images, act1320 includes acquiring “supplemental information” that reflects acorresponding pose the camera was in while the camera generated eachimage. Here, the supplemental information can optionally include imagecorrespondences.

Act 1325 then involves feeding the images and the supplemental data intoa motion model. The motion model performs temporal filtering on theimages and uses the supplemental data to generate a red-only image, agreen-only image, a blue-only image, and a monochrome image. In somecases, bilateral interpolation is performed on one or more of thered-only image, the green-only image, or the blue-only image to fill inareas that are missing color data (i.e. fill in missing colorinformation).

Act 1330 includes up-sampling the red-only image, the green-only image,and the blue-only image to cause a first resolution of the red-onlyimage, a second resolution of the green-only image, and a thirdresolution of the blue-only image to match a fourth resolution of themonochrome image. In some cases, prior to up-sampling the red-only,green-only, and blue-only images, the monochrome image might also beup-sampled or interpolated to increase the resolution of the monochromeimage. The up-sampling can be bilateral up-sampling.

In act 1335, the embodiments generate a high resolution output colorimage by combining the up-sampled red-only image, the up-sampledgreen-only image, the up-sampled blue-only image, and the monochromeimage. A resulting resolution of the high resolution output color imageis higher than any of the first, second, or third resolutions. In somecases, the first, second, and/or third resolutions might be 1280×1024.Optionally, the first, second, and/or third resolutions might be afraction of a full resolution of the sensor of the camera. For instance,the fraction might be ¼ or perhaps 1/9 (or perhaps between ¼ and 1/9) ofthe camera sensor's resolution. In some cases, the resulting resolutionof the high resolution output color image might be 2560×2048.

The process of generating the red-only image, the green-only image, andthe blue-only image can be performed using temporal filtering. That is,multiple images are acquired over time, and then data is selectivelymerged with one another to generate a new image (i.e. the process ofmotion compensated temporal filtering). Having generated these variousimages, the embodiments can then generate the high resolution outputcolor image. This image can be generated by merging the intensitiesprovided by the monochrome image with colors provided by the red-onlyimage, the green-only image, and the blue-only image.

In performing the disclosed operations, the embodiments provide improvedimage quality to a user. As a result, the user's experience ininteracting with the computer system will be enhanced.

Example Computer/Computer Systems

Attention will now be directed to FIG. 14 which illustrates an examplecomputer system 1400 that may include and/or be used to perform any ofthe operations described herein. Computer system 1400 may take variousdifferent forms. For example, computer system 1400 may be embodied as atablet 1400A, a desktop or a laptop 1400B, a wearable device 1400C, amobile device, or any other standalone device, as represented by theellipsis 1400D. Computer system 1400 may also be a distributed systemthat includes one or more connected computing components/devices thatare in communication with computer system 1400.

In its most basic configuration, computer system 1400 includes variousdifferent components. FIG. 14 shows that computer system 1400 includesone or more processor(s) 1405 (aka a “hardware processing unit”) andstorage 1410.

Regarding the processor(s) 1405, it will be appreciated that thefunctionality described herein can be performed, at least in part, byone or more hardware logic components (e.g., the processor(s) 1405). Forexample, and without limitation, illustrative types of hardware logiccomponents/processors that can be used include Field-Programmable GateArrays (“FPGA”), Program-Specific or Application-Specific IntegratedCircuits (“ASIC”), Program-Specific Standard Products (“ASSP”),System-On-A-Chip Systems (“SOC”), Complex Programmable Logic Devices(“CPLD”), Central Processing Units (“CPU”), Graphical Processing Units(“GPU”), or any other type of programmable hardware.

As used herein, the terms “executable module,” “executable component,”“component,” “module,” “model,” or “engine” can refer to hardwareprocessing units or to software objects, routines, or methods that maybe executed on computer system 1400. The different components, modules,engines, and services described herein may be implemented as objects orprocessors that execute on computer system 1400 (e.g. as separatethreads).

Storage 1410 may be physical system memory, which may be volatile,non-volatile, or some combination of the two. The term “memory” may alsobe used herein to refer to non-volatile mass storage such as physicalstorage media. If computer system 1400 is distributed, the processing,memory, and/or storage capability may be distributed as well.

Storage 1410 is shown as including executable instructions (i.e. code1415). The executable instructions represent instructions that areexecutable by the processor(s) 1405 of computer system 1400 to performthe disclosed operations, such as those described in the variousmethods.

The disclosed embodiments may comprise or utilize a special-purpose orgeneral-purpose computer including computer hardware, such as, forexample, one or more processors (such as processor(s) 1405) and systemmemory (such as storage 1410), as discussed in greater detail below.Embodiments also include physical and other computer-readable media forcarrying or storing computer-executable instructions and/or datastructures. Such computer-readable media can be any available media thatcan be accessed by a general-purpose or special-purpose computer system.Computer-readable media that store computer-executable instructions inthe form of data are “physical computer storage media” or a “hardwarestorage device.” Furthermore, computer-readable storage media, whichincludes physical computer storage media and hardware storage devices,exclude signals, carrier waves, and propagating signals. On the otherhand, computer-readable media that carry computer-executableinstructions are “transmission media” and include signals, carrierwaves, and propagating signals. Thus, by way of example and notlimitation, the current embodiments can comprise at least two distinctlydifferent kinds of computer-readable media: computer storage media andtransmission media.

Computer storage media (aka “hardware storage device”) arecomputer-readable hardware storage devices, such as RAM, ROM, EEPROM,CD-ROM, solid state drives (“SSD”) that are based on RAM, Flash memory,phase-change memory (“PCM”), or other types of memory, or other opticaldisk storage, magnetic disk storage or other magnetic storage devices,or any other medium that can be used to store desired program code meansin the form of computer-executable instructions, data, or datastructures and that can be accessed by a general-purpose orspecial-purpose computer.

Computer system 1400 may also be connected (via a wired or wirelessconnection) to external sensors (e.g., one or more remote cameras) ordevices via a network 1420. For example, computer system 1400 cancommunicate with any number devices (e.g., device 1425) or cloudservices to obtain or process data. In some cases, network 1420 mayitself be a cloud network. Furthermore, computer system 1400 may also beconnected through one or more wired or wireless networks 1420 toremote/separate computer systems(s) that are configured to perform anyof the processing described with regard to computer system 1400.

A “network,” like network 1420, is defined as one or more data linksand/or data switches that enable the transport of electronic databetween computer systems, modules, and/or other electronic devices. Wheninformation is transferred, or provided, over a network (eitherhardwired, wireless, or a combination of hardwired and wireless) to acomputer, the computer properly views the connection as a transmissionmedium. Computer system 1400 will include one or more communicationchannels that are used to communicate with the network 1420.Transmissions media include a network that can be used to carry data ordesired program code means in the form of computer-executableinstructions or in the form of data structures. Further, thesecomputer-executable instructions can be accessed by a general-purpose orspecial-purpose computer. Combinations of the above should also beincluded within the scope of computer-readable media.

Upon reaching various computer system components, program code means inthe form of computer-executable instructions or data structures can betransferred automatically from transmission media to computer storagemedia (or vice versa). For example, computer-executable instructions ordata structures received over a network or data link can be buffered inRAM within a network interface module (e.g., a network interface card or“NIC”) and then eventually transferred to computer system RAM and/or toless volatile computer storage media at a computer system. Thus, itshould be understood that computer storage media can be included incomputer system components that also (or even primarily) utilizetransmission media.

Computer-executable (or computer-interpretable) instructions comprise,for example, instructions that cause a general-purpose computer,special-purpose computer, or special-purpose processing device toperform a certain function or group of functions. Thecomputer-executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, or evensource code. Although the subject matter has been described in languagespecific to structural features and/or methodological acts, it is to beunderstood that the subject matter defined in the appended claims is notnecessarily limited to the described features or acts described above.Rather, the described features and acts are disclosed as example formsof implementing the claims.

Those skilled in the art will appreciate that the embodiments may bepracticed in network computing environments with many types of computersystem configurations, including personal computers, desktop computers,laptop computers, message processors, hand-held devices, multi-processorsystems, microprocessor-based or programmable consumer electronics,network PCs, minicomputers, mainframe computers, mobile telephones,PDAs, pagers, routers, switches, and the like. The embodiments may alsobe practiced in distributed system environments where local and remotecomputer systems that are linked (either by hardwired data links,wireless data links, or by a combination of hardwired and wireless datalinks) through a network each perform tasks (e.g. cloud computing, cloudservices and the like). In a distributed system environment, programmodules may be located in both local and remote memory storage devices.

The present invention may be embodied in other specific forms withoutdeparting from its characteristics. The described embodiments are to beconsidered in all respects only as illustrative and not restrictive. Thescope of the invention is, therefore, indicated by the appended claimsrather than by the foregoing description. All changes which come withinthe meaning and range of equivalency of the claims are to be embracedwithin their scope.

1. A method for using motion data to generate a high resolution outputcolor image from a plurality of images having sparse color information,said method comprising: causing a camera to generate a plurality ofimages, wherein a sensor of the camera has a sparse Bayer patterncomprising one or more red pixels, one or more green pixels, one or moreblue pixels, and a plurality of monochrome pixels; while the camera isgenerating the plurality of images, acquiring corresponding inertialmeasurement unit (IMU) data for each image included in the plurality ofimages such that a plurality of IMU data is also acquired, the IMU datafor each image indicating a corresponding pose the camera was in whilethe camera generated said each image; feeding the plurality of imagesand the plurality of IMU data into a motion model, wherein the motionmodel performs temporal filtering on the plurality of images and usesthe plurality of IMU data to generate a red-only image, a green-onlyimage, and a blue-only image; and generating a high resolution outputcolor image by combining the red-only image, the green-only image, andthe blue-only image.
 2. The method of claim 1, wherein each image in theplurality of images includes sparse color data and sparse intensitydata, and wherein an exposure setting of the camera remains unchangedwhile the camera generates the plurality of images.
 3. The method ofclaim 1, wherein generating the red-only image is performed by isolatingred pixels included in the plurality of images from non-red pixels andby populating the red-only image with the red pixels by placing eachrespective red pixel at a corresponding red-only image coordinate withinthe red-only image, where the corresponding red-only image coordinate isdetermined using the plurality of IMU data.
 4. The method of claim 3,wherein generating the green-only image is performed by isolating greenpixels included in the plurality of images from non-green pixels and bypopulating the green-only image with the green pixels by placing eachrespective green pixel at a corresponding green-only image coordinatewithin the green-only image, where the corresponding green-only imagecoordinate is determined using the plurality of IMU data.
 5. The methodof claim 4, wherein generating the blue-only image is performed byisolating blue pixels included in the plurality of images from non-bluepixels and by populating the blue-only image with the blue pixels byplacing each respective blue pixel at a corresponding blue-only imagecoordinate within the blue-only image, where the corresponding blue-onlycoordinate is determined using the plurality of IMU data.
 6. A computersystem that uses motion data to generate a high resolution output colorimage from a plurality of images having sparse color information, saidcomputer system comprising: at least one processor; and at least onehardware storage device that stores instructions that are executable bythe at least one processor to cause the computer system to: cause acamera to generate a plurality of images, wherein a sensor of the camerais configured to have a sparse Bayer pattern comprising one or more redpixels, one or more green pixels, one or more blue pixels, and aplurality of monochrome pixels; while the camera is generating theplurality of images, acquire supplemental information that reflects acorresponding pose the camera was in while the camera generated saideach image; feed the plurality of images and the supplementalinformation into a motion model, wherein the motion model performstemporal filtering on the plurality of images and uses the supplementalinformation to generate a red-only image, a green-only image, and ablue-only image; and generate a high resolution output color image bycombining the red-only image, the green-only image, and the blue-onlyimage.
 7. The computer system of claim 6, wherein the plurality ofmonochrome pixels is more than a sum of the one or more red pixels, theone or more green pixels, and the one or more blue pixels.
 8. Thecomputer system of claim 6, wherein a resolution of the high resolutionoutput color image is higher than resolutions of any of the imagesincluded in the plurality of images.
 9. The computer system of claim 6,wherein the camera is a low light camera, and wherein the plurality ofimages is a plurality of low light images.
 10. A method for using motiondata to generate a high resolution output color image from a pluralityof images having sparse color information, said method comprising:causing a camera to generate a plurality of images over at least aperiod of time, wherein a sensor of the camera has a sparse Bayerpattern, which includes one or more red pixels, one or more greenpixels, one or more blue pixels, and a plurality of monochrome pixels;during the period of time while the camera is generating the pluralityof images, acquiring corresponding inertial measurement unit (IMU) datafor each image included in the plurality of images such that a pluralityof IMU data is acquired; feeding the plurality of images and theplurality of IMU data into a motion model, wherein the motion model isconfigured to: generate a red-only image by isolating red pixelsincluded in the plurality of images from non-red pixels and bypopulating the red-only image with the red pixels by placing eachrespective red pixel at a corresponding red-only image coordinate withinthe red-only image, where the corresponding red-only image coordinate isdetermined using the plurality of IMU data, generate a green-only imageby isolating green pixels included in the plurality of images fromnon-green pixels and by populating the green-only image with the greenpixels by placing each respective green pixel at a correspondinggreen-only image coordinate within the green-only image, where thecorresponding green-only image coordinate is determined using theplurality of IMU data, and generate a blue-only image by isolating bluepixels included in the plurality of images from non-blue pixels and bypopulating the blue-only image with the blue pixels by placing eachrespective blue pixel at a corresponding blue-only image coordinatewithin the blue-only image, where the corresponding blue-only imagecoordinate is determined using the plurality of IMU data; and generatinga high resolution output color image by combining the red-only image,the green-only image, and the blue-only image.
 11. The method of claim10, wherein the motion model is further configured to: generate amonochrome image by isolating monochrome pixels included in theplurality of images from non-monochrome pixels and by populating themonochrome image with the monochrome pixels by placing each respectivemonochrome pixel at a corresponding monochrome image coordinate withinthe monochrome image, where the corresponding monochrome imagecoordinate is determined using the plurality of IMU data.
 12. The methodof claim 11, wherein the high resolution output color image is generatedby merging intensities provided by the monochrome image with colorsprovided by the red-only image, the green-only image, and the blue-onlyimage.
 13. The method of claim 11, wherein a resolution of themonochrome image is higher than resolutions of any of the images in theplurality of images.
 14. The method of claim 13, wherein a resolution ofthe green-only image is higher than the resolutions of the plurality ofimages.
 15. The method of claim 14, wherein a resolution of theblue-only image is higher than the resolution of the plurality ofimages.
 16. The method of claim 11, wherein the high resolution outputcolor image is generated by merging intensities provided by themonochrome image with colors provided by the red-only image, thegreen-only image, and the blue-only image, and wherein an intensity ofthe high resolution output color image is higher than an intensity ofany of the red-only image, the green-only image, or the blue-only image.17. The method of claim 10, wherein combining the red-only image, thegreen-only image, and the blue-only image is performed by merging redpixels from the red-only image with corresponding green pixels from thegreen-only image and with corresponding blue pixels from the blue-onlyimage.
 18. (canceled)
 19. The method of claim 10, wherein the sensor,which is configured to have the sparse Bayer pattern, includes one redpixel, two green pixels, one blue pixel, and twelve monochrome pixels.20. The method of claim 10, wherein a red sensor channel of the sensorisolates the red pixels from the non-red pixels.
 21. The method of claim1, wherein the temporal filtering includes stacking the plurality ofimages on top of one another and combining data from the images in theplurality of images, and wherein, prior to stacking the plurality ofimages, poses of the plurality of images are aligned with one another.